[Single File] Add single file support for Flux Transformer#9083
Conversation
DN6
changed the title
| mlp_ratio = 4.0 | ||
| inner_dim = 3072 |
There was a problem hiding this comment.
Should be okay to hardcode these no? They are the same across both models. We can grab inner_dim from the checkpoint, but not sure about mlp_ratio?
There was a problem hiding this comment.
I think okay to hard code. I would just define them as proper constants at the top of the file.
There was a problem hiding this comment.
But if they're only applied within the scope of this function, they don't need to exist as constants that can be accessed globally?
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
| mlp_ratio = 4.0 | ||
| inner_dim = 3072 | ||
|
|
||
| # in SD3 original implementation of AdaLayerNormContinuous, it split linear projection output into shift, scale; |
There was a problem hiding this comment.
Can leave outside of the function no since SD3 also uses it?
sayakpaul
left a comment
sayakpaul
left a comment
There was a problem hiding this comment.
LGTM!
Two things:
- Let's add it to the list of models that support single file loading.
- Let's see if FP8 support works? If so, let's attach a code snippet?
|
@DN6 let's merge this after the TODOs? |
* update * update * update --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
3 participants
What does this PR do?
Add single file support for the Flux Transformer model.
Fixes # (issue)
Before submitting
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.